Testing software systems belonging to organizations is often a time consuming and expensive process. This is especially true when new software modules are installed or when software modules undergo changes such as when they are updated and/or customized. Such software systems are often large and complex and require the running of many tests for their validation. Since different organizations often utilize many software modules that are the same or similar, they end up running many similar tests. Were the different organizations able to share their accumulated testing-related knowledge, which may be considered testing crowd wisdom, they might benefit from that immensely. For example, knowledge such as best practices and/or system vulnerabilities may not need to be discovered by each organization separately. Once certain tests have been shown to be useful for many organizations, they might be suggested to other organizations.
However, despite the fact that different organizations often end up utilizing similar tests, it is often the case that each organization develops its own testing suite. Software systems belonging to different organizations are often each customized for their respective organizations. Thus, it is not likely that a test devised for a first organization will run “as is” on a system belonging to a second organization. Additionally, tests for a system belonging to an organization often involve proprietary data related to the organization. Thus, organizations are not likely to share their tests if it means that in the process, their proprietary data is at risk of being leaked to an unauthorized party.
Moreover, even if organizations share their testing data, there still may be a problem to select and/or generate tests from the large body of testing data that may be available. Given that there may be many different tests for many software modules, it is often far from trivial to be able to effectively leverage testing data collected from different organizations for the needs of a certain user or a certain organization.